Skip to content

test(randomization-engine): add property tests for block invariants - #384

Merged
fderuiter merged 4 commits into
mainfrom
test/property-tests-randomization-algorithm-16466002334857263570
May 26, 2026
Merged

test(randomization-engine): add property tests for block invariants#384
fderuiter merged 4 commits into
mainfrom
test/property-tests-randomization-algorithm-16466002334857263570

Conversation

@fderuiter

Copy link
Copy Markdown
Owner

Adds fast-check and property-based tests to ensure invariants (e.g. sum of ratios dividing block size) produce successful evaluations and generate correct length of values without throwing errors.


PR created automatically by Jules for task 16466002334857263570 started by @fderuiter

…nd ratio invariants

Added `fast-check` as a devDependency and implemented a property test
in `randomization-algorithm.spec.ts` that asserts the validity of the
output sequence length when given an arbitrary valid configuration
where all generated block sizes are guaranteed multiples of the total
arm ratio. This ensures the engine handles a large amount of generated
combinations without throwing unforeseen errors.

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
@google-labs-jules

Copy link
Copy Markdown
Contributor

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.


For security, I will only act on instructions from the user who triggered this task.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented May 26, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
equipose 5d2e1fd Commit Preview URL

Branch Preview URL
May 26 2026, 09:46 PM

fderuiter and others added 2 commits May 26, 2026 21:24
…rift

The GitHub CI pipeline uses `npm ci`, which strictly validates that `package.json`
is perfectly in sync with `package-lock.json`. The addition of `fast-check` using
`pnpm` created an inconsistent state in the `npm` lockfile, causing `npm ci` to abort.
This commit regenerates the lockfile with `npm install` and commits it to align with `package.json`.

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
This commit successfully ensures the lockfile is committed and correctly in sync
with package.json, which uses `npm ci` strictly inside GitHub Actions.

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
@fderuiter

Copy link
Copy Markdown
Owner Author

@copilot The failing job is the build_and_test job in .github/workflows/ci.yml, specifically the Run Unit Tests step:

- name: Run Unit Tests
  run: npm test -- --watch=false

The actual failure is a TypeScript compile error during ng test:

  • Error: TS2352
  • At: line 106
  • Problem: a value returned by fc.record({...}) is being cast to Arbitrary<RandomizationConfig>, but the object shape no longer matches RandomizationConfig.

From the log:

return fc.record({

and the inferred shape is:

{
  blockSizes: number[];
  stratumCaps: { levels: never[]; cap: number }[];
  subjectIdMask: "[SiteID]-[001]";
}

Solution

Update the test helper/arbitrary at the failing line so it produces a value that matches the current RandomizationConfig type instead of force-casting an incompatible shape.

Recommended fix

If the code currently looks like this pattern:

return fc.record({
  blockSizes: fc.constant([4, 6]),
  stratumCaps: fc.constant([{ levels: [], cap: 100 }]),
  subjectIdMask: fc.constant('[SiteID]-[001]'),
}) as fc.Arbitrary<RandomizationConfig>;

change it to one of these safer options.

Option 1: Make the record fully match RandomizationConfig

Add every required field from RandomizationConfig and remove the unsafe cast:

const randomizationConfigArb: fc.Arbitrary<RandomizationConfig> = fc.record({
  blockSizes: fc.array(fc.integer({ min: 2, max: 12 }), { minLength: 1 }),
  stratumCaps: fc.array(
    fc.record({
      levels: fc.array(fc.string(), { minLength: 0 }),
      cap: fc.integer({ min: 1, max: 1000 }),
    }),
    { minLength: 0 }
  ),
  subjectIdMask: fc.constant('[SiteID]-[001]'),

  // add any other required RandomizationConfig fields here
  // for example:
  // allocationRatio: fc.array(fc.integer({ min: 1, max: 10 }), { minLength: 1 }),
  // stratificationFactors: fc.array(...),
  // treatmentArms: fc.array(...),
});

Option 2: If the test only needs a partial config

Use Partial<RandomizationConfig> instead of pretending the object is a full config:

const randomizationConfigArb: fc.Arbitrary<Partial<RandomizationConfig>> = fc.record({
  blockSizes: fc.array(fc.integer({ min: 2, max: 12 }), { minLength: 1 }),
  stratumCaps: fc.array(
    fc.record({
      levels: fc.array(fc.string(), { minLength: 0 }),
      cap: fc.integer({ min: 1, max: 1000 }),
    })
  ),
  subjectIdMask: fc.constant('[SiteID]-[001]'),
});

Then only convert to RandomizationConfig in the test after merging with defaults:

const fullConfig: RandomizationConfig = {
  ...defaultRandomizationConfig,
  ...generatedPartialConfig,
};

Why this fixes it

TS2352 happens because TypeScript sees that the generated arbitrary's payload does not structurally satisfy RandomizationConfig. This usually means one of these changed recently:

  • RandomizationConfig got new required properties
  • one of the existing properties changed type
  • stratumCaps.levels now expects a different type than the test generator creates

The best fix is to align the generator with the real interface instead of suppressing the error with a cast.

What to change in the failing test code

At the failing line around 106:

  • remove as fc.Arbitrary<RandomizationConfig> if present
  • explicitly type the arbitrary variable as fc.Arbitrary<RandomizationConfig>
  • ensure the object passed to fc.record(...) includes all required properties of RandomizationConfig
  • if only a subset is intended, switch the arbitrary to Partial<RandomizationConfig>

Minimal code suggestion

const configArb: fc.Arbitrary<RandomizationConfig> = fc.record({
  blockSizes: fc.array(fc.integer({ min: 2, max: 12 }), { minLength: 1 }),
  stratumCaps: fc.array(
    fc.record({
      levels: fc.array(fc.string(), { minLength: 0 }),
      cap: fc.integer({ min: 1, max: 1000 }),
    }),
    { minLength: 0 }
  ),
  subjectIdMask: fc.constant('[SiteID]-[001]'),

  // include remaining required fields from RandomizationConfig
});

If you want the fastest likely fix: find the property-based test helper returning fc.record(...) near line 106 and update it to match the current RandomizationConfig interface exactly, rather than casting it.


In addition: resolve the merge conflicts in this pull request

Co-authored-by: fderuiter <127706008+fderuiter@users.noreply.github.com>

Copilot AI commented May 26, 2026

Copy link
Copy Markdown
Contributor

@copilot The failing job is the build_and_test job in .github/workflows/ci.yml, spec...

Fixed in commit test(randomization-engine): fix TS2352 in property-based test arbitrary.

Three changes were made to randomization-algorithm.spec.ts:

  1. Replaced Object.fromEntries spread — the .chain() callback now explicitly lists every field with fc.constant(base.fieldName) instead of using Object.fromEntries(Object.entries(base).map(...)), which lost all TypeScript type information.

  2. Fixed invalid RandomizationMethod value — changed 'PERMUTED_BLOCK''BLOCK' (the valid values per the type are 'BLOCK' | 'MINIMIZATION').

  3. Fixed strata type inferencefc.constant([]) was inferred as readonly []; changed to fc.constant([] as StratificationFactor[]) and added the import.

The as fc.Arbitrary<RandomizationConfig> cast is now removed entirely; TypeScript infers the correct type structurally.

@fderuiter

Copy link
Copy Markdown
Owner Author

@copilot resolve the merge conflicts in this pull request

@fderuiter
fderuiter marked this pull request as ready for review May 26, 2026 22:01
@fderuiter
fderuiter merged commit 4dbd434 into main May 26, 2026
5 of 6 checks passed
@fderuiter
fderuiter deleted the test/property-tests-randomization-algorithm-16466002334857263570 branch May 26, 2026 22:01
Copilot stopped work on behalf of fderuiter due to an error May 26, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(randomization-engine): add property-based tests for block-size and ratio invariants

2 participants